Method and system for producing interactive, three-dimensional renderings of selected body organs having hollow lumens to enable simulated movement through the lumen

ABSTRACT

A method and system are provided for effecting interactive, three-dimensional renderings of selected body organs for purposes of medical observation and diagnosis. A series of CT images of the selected body organs are acquired. The series of CT images is stacked to form a three-dimensional volume file. To facilitate interactive three-dimensional rendering, the three-dimensional volume file may be subjected to an optional dataset reduction procedure to reduce pixel resolution and/or to divide the three-dimensional volume file into selected subvolumes. From a selected volume or subvolume, the image of a selected body organ is segmented or isolated. A wireframe model of the segmented organ image is then generated to enable interactive, three-dimensional rendering of the selected organ.

RELATED APPLICATIONS

This application is a continuation of application Ser. No. 11/157,136,entitled “Method and System for Producing Interactive, Three-DimensionalRendering of Selected Body Organs Having Hollow Lumens to EnableSimulated Movement Through the Lumen”, filed Jun. 20, 2005, now U.S.Pat. No. 7,792,565 which is a continuation of issued application Ser.No. 09/898,141, now U.S. Pat. No. 6,909,913, filed Jul. 3, 2001, whichis a continuation of issued patent application Ser. No. 08/734,427,filed on Oct. 16, 1996, now U.S. Pat. No. 6,272,366, which is acontinuation of issued patent application Ser. No. 08/331,352, filed onOct. 27, 1994, now U.S. Pat. No. 5,782,762, which are incorporatedherein by reference.

BACKGROUND OF THE INVENTION

For many forms of cancer, early detection is essential for a favorableprognosis. The cancerous growth must be detected at an early stagebefore the cancer is allowed to grow and spread. Such is the case forcolorectal and lung cancers. As a result, techniques have been developedto examine the colon and tracheobronchial airways for the growth ofprecancerous and cancerous masses.

Colon cancer is the second leading cause of cancer death in the UnitedStates today. Fortunately, most colorectal carcinomas arise frompreexisting adenomatous polyps, and the risk of cancer is directlyrelated to the size of the polyp (1% in polyps less than 1 cm, 10% inpolyps between 1 and 2 cm, and 30% in polyps 2 cm or greater).Scientific studies suggest that the early detection and removal of smallcarcinomas and precursor adenomatous polyps reduces mortality.Therefore, current strategies for colon cancer screening focus on theearly discovery of polyps. The techniques used to screen for colorectalcancer include flexible sigmoidoscopy (the use of a fiberoptic scope toexamine the distal half of the colon) and fecal occult blood testing(wherein hemorrhage is detected). There is some debate on theeffectiveness of colorectal cancer screening, but it has been predictedthat a 30-40% reduction in mortality can be achieved with properscreening using a combination of fecal occult blood testing andsigmoidoscopy.

The National Cancer Institute and the American Cancer Society recommendcolorectal cancer screening for persons of average risk who are morethan 50 years old using sigmoidoscopy and annual fecal occult bloodtests. The fecal occult blood test is easy to perform but is plagued bymany false positive and false negative results. Sigmoidoscopy suffersfrom the fact that it only examines the distal half of the colon (therectum and the sigmoid colon). This is a serious shortcoming sinceapproximately 40% of colorectal carcinomas occur proximal to the splenicflexure and are therefore undetectable using sigmoidoscopy.

Examination of the entire colon by a barium enema or conventionalcolonoscopy increases sensitivity for cancer detection but alsoincreases the risks and costs. A barium enema causes patient discomfortand/or embarrassment and exposes the patient to substantial radiation.Colonoscopy, like sigmoidoscopy, does not always examine the entirecolon since the cecum is not reached in approximately 15% ofcolonoscopies. In addition, colonoscopy requires patient sedation,places the patient at risk for bowel perforation, and is comparativelyexpensive. Furthermore, with the exception of fecal occult bloodtesting, all of these procedures meet with significant patientdiscomfort.

Turning now to the tracheobronchial examination, Transbronchial NeedleAspiration (TBNA) is a bronchoscopy technique that permits theoutpatient diagnosis and staging of mediastinal disease. This procedureallows for the outpatient sampling of tissue specimens that mightotherwise require a surgical procedure. With TBNA, a needle is placedthrough an airway wall in the vicinity of a suspected lesion to retrievea tissue sample. Conventionally, the bronchoscopist is guided only by amental model of the patient's anatomy and pathology following review ofbronchoscopy images and/or a series of thoracic computed tomography (CT)images. As can be expected, proper placement of the needle can beextremely difficult and to a small degree somewhat imprecise.

Accordingly, it is highly desirable to have a reliable, efficient methodfor examining the tracheobronchial tree and/or the colon of a patient todetect early cancer. The technique should allow for the discovery ofpolyps of 1 cm or greater in size in the colon and 5 mm or greater inthe airways. Preferably, the method should reduce the amount ofdiscomfort encountered by the patient, decrease the risk of injury tothe patient, and be conducted in a reasonable amount of time withoutbeing prohibitively expensive. Preferably, the method should benon-invasive or minimally invasive.

SUMMARY OF THE INVENTION

In accordance with the present invention, a system and method areprovided for producing two-dimensional images of a selected structure,such as a portion of the body, to enable the creation of athree-dimensional rendering of the selected structure. Morespecifically, two-dimensional images of a selected body portion areacquired with use of a scanner, for example, a helical computedtomography (CT) scanner. The two-dimensional images are then stacked tocreate a three-dimensional image volume. From the three-dimensionalvolume, image features of one or more selected body organs are isolatedor segmented. Isosurfaces of the segmented organs are produced andwireframe models are then generated from each of the isosurfaces for therespective segmented organs. The wireframe models are used to generatereal time, three-dimensional images (renderings) of the selected organs.

In a specific application for generating a three-dimensional renderingof a patient's colon, the patient initially undergoes a selectedpreparation procedure. For example, the patient's colon is initiallycleansed and then inflated with air to permit the acquisition ofunobstructed two-dimensional images of the colon. Next, the patientundergoes a CT scan to produce a series of two-dimensional images of thepatient's internal organs. Preferably, a spiral or helical CT scanner isemployed to provide a series of uninterrupted two-dimensional imagesthrough the body. The series of two-dimensional images are transferredfrom the scanner to a graphics computer to effect various imageprocessing procedures. The dataset corresponding to the series oftwo-dimensional images may be transferred to the graphics computer in acompressed format for decompression on the graphics computer.Alternatively, the dataset representing the series of two-dimensionalimages may be decompressed on the computer console of the scanner priorto transfer to the graphics computer.

After transfer to the graphics computer, the series of two-dimensionalimages are stacked in order to form a three-dimensional volume file. Inorder to facilitate three-dimensional rendering of a selected organcontained within the three-dimensional volume of images, thethree-dimensional volume file may be subjected to various optionaldataset reduction techniques. For example, a reduction of pixelresolution on the series of two-dimensional images may be effected. Inaddition, the three-dimensional volume file may be separated intoselected subvolumes.

After the optional dataset reduction procedure is completed, an imagesegmentation process is performed in order to isolate features of aselected organ or region of interest from the three-dimensional volumefile. Image segmentation may be effected by various techniques. Forexample, an image slice through the three-dimensional volume file may besubjected to a thresholding process in which a physical property of thetwo-dimensional image slice, such as x-ray attenuation, may be used toestablish a particular threshold range, such as a range of x-rayattenuation values, that corresponds to the organ of interest. After anappropriate threshold range is determined, the entire three-dimensionalvolume file is then thresholded to segment the organ of interest. Forexample, in order to segment the colon, a threshold range correspondingto the air column within the colon could be selected to isolate theinner wall of the colon.

An alternative segmentation technique may be employed in which a regiongrowing technique is used to isolate the air column within the colon.Using the region growing technique, a “seed” is planted by selecting adata point or voxel within the air column of the colon. Neighboringvoxels are progressively tested for compliance with a selectedacceptance criteria, such as x-ray attenuation values falling within aselected threshold range representing air. As such, the seed regioncontinues to expand or grow until the entire air column within the lumenof the colon is filled.

A surface, or isosurface, of the air column representing the colon isthen produced. A wireframe model of the isosurface is then generatedusing a selected image processing technique such as a marching cubesalgorithm. From the wireframe model of the colon, a three-dimensionalinteractive rendering is produced that enables the user to rapidly viewa series of three-dimensional images of the lumen of the colon forpurpose of detection of pathological conditions. As such, a user has theperception of flying through the lumen and virtual reality environmentis achieved.

Other procedures may be employed with the interactive, three-dimensionalrendering of the organ for purposes of medical diagnosis or observation.For example, a three-dimensional rendering of the colon may be splitopen so that the interior half of the colon along a selected length maybe viewed. In addition, selected flights or paths of movement throughthe lumen of the colon may be recorded on videotape or, alternatively,stored in the memory of the graphics computer as a path of coordinatesfor subsequent reproduction and display. In addition, various areas ofpathology may be marked by different colors during a flight through thelumen of the colon to facilitate subsequent detection.

The method and system of the present invention may also be used toisolate other organs. For example, the system and method may be used toisolate the tracheobronchial airways, as well as the surrounding lymphnodes and blood vessels. During an interactive, three-dimensionalrendering, a selected organ may be rendered transparent to facilitate orpermit a view of the remaining (surrounding/neighboring) organs. Forexample, the airway walls may be made transparent so that thesurrounding blood vessels and lymph nodes become visible from aperspective within the interior of the airways.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing summary, as well as the following detailed description ofthe preferred embodiments of the present invention, will be betterunderstood when read in conjunction with the accompanying drawings, inwhich:

FIG. 1 is a flowchart representing a method in accordance with thepresent invention of producing interactive, three-dimensional renderingsof selected structures such as a selected body organ;

FIG. 2 is a flowchart representing a general process in accordance withthe present invention of converting two-dimensional images of a selectedbody organ into a three-dimensional image;

FIG. 3 is a block diagram of a system used in the method of the presentinvention;

FIG. 4 is a flowchart representing the steps involved in a datareduction process used in the method of the present invention;

FIG. 5 is a flowchart representing the steps involved in segmentation ofan image of the selected body organ from a three-dimensional volume;

FIG. 6 is a flowchart representing the steps involved in a regiongrowing procedure for segmentation of an image of the selected bodyorgan from a three-dimensional volume;

FIG. 7 is a perspective view of a three-dimensional rendering showingthe selection of coordinates X_(min), X_(max), Y_(min), Y_(max),Z_(min), and Z_(max) to define a selected subvolume containing a colon;

FIG. 8 is a view of a thresholded two-dimensional image slice of a colondisplayed together with the corresponding gray-scale image;

FIG. 9 is a representation of a wireframe model of a selected portion ofthe colon;

FIG. 10 is a three-dimensional rendering of the portion of the colonrepresented by the wireframe model shown in FIG. 9;

FIG. 11 is an external perspective view of a three-dimensional renderingof a selected portion of the colon;

FIG. 12 is a split-open view of the selected portion of the colon shownin FIG. 11;

FIG. 13 is a map view showing a three-dimensional rendering of the colonon which a location marker is superimposed on the image;

FIG. 14 is a three-dimensional rendering of a selected section of thecolon corresponding to the location of the marker shown in FIG. 13;

FIG. 15 is a perspective view of a three-dimensional rendering showingthe selection of coordinates X_(min), X_(max), Y_(min), Y_(max),Z_(min), and Z_(max) to define a selected subvolume containing atracheobronchial tree;

FIG. 16 is a view of a thresholded two-dimensional image slice of atracheobronchial tree displayed together with the correspondinggray-scale image;

FIG. 17 is a representation of a wireframe model of a selected portionof the tracheobronchial tree;

FIG. 18 is a three-dimensional rendering of the portion of thetracheobronchial tree represented by the wireframe model shown in FIG.17;

FIG. 19 is an external perspective view of a three-dimensional renderingof a selected portion of the tracheobronchial tree;

FIG. 20 is a split-open view of the selected portion of thetracheobronchial tree shown in FIG. 19;

FIG. 21 is a map view showing a three-dimensional rendering of thetracheobronchial tree on which a location marker is superimposed on theimage;

FIG. 22 is a three-dimensional rendering of a selected section of thetracheobronchial tree corresponding to the location of the marker shownin FIG. 21;

FIG. 23 is a diagrammatic view depicting a model for determining acenter line through a lumen;

FIG. 24 is a flowchart representing a procedure for making athree-dimensional rendering of a selected organ transparent relative tothree-dimensional renderings of remaining organs;

FIG. 25 is a flowchart representing a procedure for establishing “go to”points at selected locations on a three-dimensional rendering to causethe display of a three-dimensional rendering of the specific locationselected by a respective “go to” point; and

FIG. 26 is a diagram depicting a pick point procedure in which aselected pick point on a three-dimensional image causes the display ofthree orthogonal planes passing through the pick point.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention generally relates to a method and system, asschematically represented in FIGS. 1, 2, and 3, for generating anddisplaying interactive, three-dimensional structures. Thethree-dimensional structures are in the general form of selected regionsof the body and, in particular, body organs with hollow lumens such ascolons, tracheobronchial airways, blood vessels, and the like. Inaccordance with the method and system of the present invention,interactive, three-dimensional renderings of a selected body organ aregenerated from a series of acquired two-dimensional images.

As illustrated in FIG. 3, a scanner 22, such as a spiral or helical CT(Computed Tomography) scanner, operated by a computer console 24 is usedto scan a selected three-dimensional structure, such as a selectedanatomy, thereby generating a series of two-dimensional images 12through that structure. A general procedure for converting ortransforming the set of two-dimensional images 12 of the structure intoa three-dimensional rendered image 17 is schematically illustrated inFIG. 2. Any type of digital image (CT, MR, US, SPECT, PET) is amendableto three-dimensional image processing. However, the acquisition ofcontiguous thin slices is imperative for generating acceptablethree-dimensional images. Recent developments of volume acquisitiondevices such as spiral/helical CT, three-dimensional MRI, andthree-dimensional ultrasound equipment make it feasible to renderthree-dimensional images of any organ system.

Referring to FIG. 2, each of the individual two-dimensional images 12defines a two-dimensional (X-Y) matrix of picture elements, or pixels,with each pixel representing a predetermined physical propertyassociated with the three-dimensional structure or organ at a particularlocation within the three-dimensional structure. Successivetwo-dimensional images 12 are spaced in a third-dimensional direction(Z) throughout the three-dimensional structure. The two-dimensionalimages 12 are typically obtained from a helical computed tomography (CT)scanner 22 operated by a computer console 24. For example, the scannermay be a General Electric HiSpeed Advantage Helical CT Scanner connectedwith an optional General Electric Independent computer console orphysician's console. The computer console 24 may, however, be anintegral part of the helical CT scanner 22 instead of a separateindependent console. The two-dimensional images 12 can also be obtainedfrom ultrasound, positron emission tomography, emission computedtomography, and magnetic resonance imaging. The physical propertymeasured is directly associated with the scanning technique used toproduce the two-dimensional images 12. For CT images the physicalproperty measured is x-ray attenuation, while for magnetic resonanceimages (MRI) the physical property measured is related to variousproperties such as proton density.

As shown in FIG. 2, the series of two-dimensional images 12 is stackedto form a three-dimensional volume 13, thereby defining athree-dimensional matrix (X, Y, Z axes) that represents at least onephysical property associated with the three-dimensional structure atcoordinates positioned throughout the three-dimensional volume. Thethree-dimensional matrix is composed of three-dimensional volumeelements, or voxels, which are analogous to two-dimensional pixels. Fromthe three-dimensional volume 13, a targeted volume 14 is selected forthree-dimensional rendering. For example, the targeted volume 14 mayinclude a selected organ or region of interest that is to be isolatedfrom the original three-dimensional volume 13. The targeted volume 14may include an entire organ or, alternatively, may include an air columnor volume confined within an organ or a lumen of the organ.

A dataset representing the original three-dimensional volume 13 may beoptionally subjected to a dataset reduction process to decrease image orspatial resolution or to divide the original volume into smallersubvolumes of data prior to isolating the targeted volume 14. Datasetreduction and/or subvolume selection are only necessary if thecapabilities of the graphics computer 26 are inadequate to process thefull three-dimensional volume 13 for effective or efficientthree-dimensional rendering. In order to obtain three-dimensionalconstructions 17 in real time, it may be necessary to reduce the size ofthe dataset representing the three-dimensional volume 13 of images. Forexample, the dataset of the original three-dimensional volume 13 mayneed to be reduced from an original size of 100-250 Megabytes to areduced size of about 5-10 Megabytes. However, the amount of reduction,if any, may vary depending upon the size of the original dataset and theprocessing speed and capability of the graphics computer 26 used forthree-dimensional rendering.

An optional dataset reduction process 65 is generally represented inFIG. 4. When necessary, dataset reduction can be accomplished byreducing the pixel resolution, for example, from 16 bits per pixel to 8bits per pixel. The reduction in pixel resolution, represented by step67 in FIG. 4, decreases the image contrast in the final displayedthree-dimensional images by reducing the number of shades of gray from2¹⁶ to 2⁸. While not preferable in particular applications, datasetreduction may also be effected by decreasing spatial resolution, asrepresented by step 69 in FIG. 4. For example, each two-dimensionalimage 12 stored on a matrix of 512 pixels×512 pixels may be reduced to amatrix of 256 pixels×256 pixels through spatial manipulation ofinformation.

Further reduction of the dataset size of the original three-dimensionalvolume 13 can be effected, if necessary, by selecting a subvolume of thedataset to be displayed, as represented by step 68 of FIG. 4. Forexample, the colon can be subdivided into the rectum, the sigmoid colon,the descending colon, the splenic flexure, the transverse colon, thehepatic flexure, the ascending colon, and the cecum. A separatesubvolume may then be created for one or more of such subdivisions ofthe colon.

Following dataset reduction, the three-dimensional image of the organ orregion of interest is segmented (isolated) from the volume of data. Arange of tissue values, which depends on the physical property measured(e.g., x-ray attenuation), may be selected to designate the organ orregion of interest. The organ or region of interest is then isolatedfrom the volume of data. A general method or process 70 for segmentingan image of a selected organ or region of interest is represented inFIG. 5. The region of interest may, for example, be the air columncomprising the colon or tracheobronchial airways, or some other bodypart having a lumen filled with a homogeneous substance (i.e., air,blood, urine, etc.). Alternatively, the region of interest may be asection of bone.

The segmentation process 70 may be effected, for example, by designatinga range of physical property values bounded by threshold limits thatfunction to designate the organ of interest. For the selected volume orsubvolume, voxels falling within a selected thresholding range areassigned a single value, for example, 255 to provide a white color,whereas voxels falling outside of the selected thresholding range areassigned a different single value, for example, 0 to provide a blackcolor. Accordingly, the selected volume is thresholded by comparing eachvoxel in the selected volume to the threshold limits and by assigningthe appropriate color value 0 or 255 to each such voxel depending onwhether each such voxel falls inside or outside the threshold rangedefined by the threshold limits. By thresholding the selected volume,the target volume 14 is formed having equal voxel values. Morespecifically, a target organ 14 is produced having a white color whileall volumes outside the threshold limits are produced having a blackcolor. With the exception of various artifacts, which can be eliminatedusing image-processing techniques, only the thresholded target organ 14is colored white and everything else is colored black.

From the thresholded target volume 14, an isosurface 15 is defined. Theisosurface 15 is an exterior surface of equal voxel value on thethresholded target volume 14 which is established by the selectedthresholding range.

A wireframe model 16 is generated from the isosurface 15. The wireframemodel 16 is formed as a series of polygonal surfaces that approximatesthe surface of the region of interest such as the selected organ. Thewireframe model 16 is defined by a series of vertices which areinterconnected by a set of line segments. The wireframe model 16 appearsas a three-dimensional wire mesh object which can be rendered into athree-dimensional image 17. The three-dimensional image 17 is generatedby appropriately shading the polygons of the wireframe model 16 toprovide a three-dimensional image of the selected organ. Thethree-dimensional image 17 is displayed on a computer monitor 28.Additionally, the displayed imagery 17 can be recorded on a videorecorder 30 or photographed for future viewing. An input, in the form ofa computer mouse 27, is provided on the graphics computer 26 to permit auser to manipulate the displayed imagery.

The method of the present invention can be used to display a colon inthree dimensions and, in a more specific application, to enablereal-time or interactive three-dimensional rendering of the colon whichthereby enables user interaction with the colon imagery, i.e. virtualreality. While user interaction with the three-dimensional images of thecolon is simulated, the three-dimensional image itself is an accurate,three-dimensional view of the actual colon.

A method for generating interactive, three-dimensional renderings of apatient's colon in accordance with the present invention is generallyset forth in FIG. 1. At step 40, a patient is initially prepared forimaging by cleansing the patient's colon. The cleansing procedure can beaccomplished with a clear liquid diet in conjunction with laxatives.Alternatively, a Golytely prep can be administered on the day before theexam. The purpose of the cleansing procedure is to eliminate feces fromthe colon. Optimally, an absolutely clean colon is desired prior tocomputed tomography (CT) scanning. Any retained feces or fluid cansimulate or mask small polyps because it is sometimes difficult todifferentiate feces from the colon wall. The effectiveness of using aclear liquid diet or a Golytely prep procedure may still be somewhathindered by small amounts of retained feces. As an alternative tocleansing the colon, or in conjunction with cleansing the colon, thepatient can be fed a low residue diet combined with a contrast agent(such as a low density barium, for example, 1.5% W/V barium) for aboutthree days. Such a procedure may serve to homogeneously opacify anyretained stool so that the image of the feces can then be subtractedfrom the final display, or at least from selected images, using imageprocessing techniques.

Once the colon has been cleansed, the colon is insufflated with gas todistend the colon. Distending the colon assures that the interiorsurface of the colon will be clearly visible in the final image display.A rectal catheter, i.e., a standard barium enema catheter (0.5 inch(1.27 cm) diameter is inserted and gas is introduced into the colon. Thecolon is filled to a predetermined pressure or volume by introducing thegas either as discrete puffs or at a constant flow rate. Although aircan be used, CO₂ may be preferred since CO₂ passes through the colonicmucosa into the bloodstream and is subsequently exhaled, therebydecreasing the amount of bloating and cramping experienced by a patientafter the examination. Unlike conventional colonoscopy, there is no needto sedate the patient for this procedure.

After insufflation, the colon is then scanned, at step 45 of FIG. 1, bya helical CT scanner 22 to produce a series of two-dimensional images 12of the colon. The picture elements, or pixels, in the images 12represent at least one physical property associated with the colon. Thephysical property, for example, may be the x-ray attenuation of thecolon wall or of the air column within the colon. The images 12 aregenerally taken at regularly spaced locations throughout the abdominalregion of the patient. The smaller the spacing between successive images12, the better the resolution in the final displayed imagery.Preferably, the spacing between successive images 12 is approximately 1mm to produce isocubic voxels since the X and Y dimensions are eachapproximately 1 mm.

Immediately after inflating the colon, the abdomen is scanned, forexample, with a GE HiSpeed Advantage Helical CT Scanner 22, during asingle breath-hold acquisition which is completed in about 30-50seconds. The scanning parameters may consist of a 0.20 inch (5 mm) x-raybeam collimation, 0.4 inch/sec (10 mm/sec) table speed to provide 2:1pitch, and a 0.04 inch (1 mm) image reconstruction interval. The x-raybeam collimation is the width of the x-ray beam which therebyestablishes the CT slice thickness and Z-axis spatial resolution. Thepitch is the CT table speed divided by the collimation. The imagereconstruction interval reflects the interval at which two-dimensionalimages are reconstructed. Using an x-ray beam collimation of 5 mm and aselected reconstruction interval of 1 mm, images reflecting a 5 mm CTslice thickness are generated at 1 mm intervals. Consequently, there isan overlap of 4 mm between successive 5 mm thick images at a 1 mmreconstruction interval.

A 50 second scan at a table speed of 10 mm per second in a Z-axisdirection creates a volume of data having a Z-axis length of 500 mm.Using a 1 mm reconstruction interval over the length of the Z-axis ofthe volume of data produces 500 images with each image representing a 5mm thick section along the Z-axis. Consequently, up to 500 images mayneed to be stored in compressed format on the CT scanner console or on aGE Independent Console (physician computer console) 24 associated withthe scanner 22. After completion of the CT scan, the rectal catheter isremoved, the gas is expelled, and the patient is discharged.

The set of CT images 12 consisting of up to 500 images is thenextracted, at step 50 of FIG. 1, from a database on the computer console24 in compressed format. Once the data has been extracted, the data istransferred from the console 24 at step 52 of FIG. 1, over a fiberopticnetwork 25, to a graphics computer work station 26, such as a CrimsonVGXT computer work station (150 MHz processor, 256 Mbytes RAM) fromSilicon Graphics, Inc. (SGI, Mountain View, Calif.). The image files 12are preferably transferred in the compressed format and thendecompressed on the SGI graphics computer 26. Alternatively, the imagefiles can be decompressed on the GE computer console 24 and thentransferred to the SGI graphics computer 26. The fiberoptic network 25may comprise an ethernet network, asynchronous transfer mode (ATM)network, or a Fiber Distributed Data Interface (FDDI) network.

The extraction and transfer processes, 50 and 52, are performed by threeprogram modules. To perform the extraction process 50, the first moduleresiding on the computer console 24 extracts the images one at a timefrom the image database on the computer console and places eachextracted image file in a subdirectory on the computer console 24. Inaddition to the CT image files, a text file containing information aboutthe patient and the type of case (e.g. colon) is placed in thesubdirectory on the computer console 24 so that the extracted imagefiles are properly correlated with the appropriate patient and type ofcase. The second module, which resides on the graphics computer 26 isinitiated every 5 minutes. The purpose of the second module is totransfer the text file and corresponding image files from the computerconsole 24 to the graphics computer 26 and to delete such files from thecomputer console. The text file and image files are stored in atemporary directory on the graphics computer 26. The third module isalso initiated every 5 minutes and is interleaved with the secondmodule. The third module determines if all of the files associated witha specific patient have been transferred to the graphics computer. Ifall of the files of the specific patient have been transferred, thethird module then organizes the transferred files in a patientsubdirectory on the graphics computer 26 according to the case type. Theentire process generally takes about 1 hour and is a rate limiting step.The image transfer time can be reduced, however, by using the DICOM 3image standard for data transmission.

Once the data transfer to the graphics computer 26 is complete, thecompressed image data is then decompressed, at step 55 of FIG. 1. Thedecompression of the image data is effected in accordance with adecompression scheme that is dependent on the specific compressionformula used to originally compress the data. Differentcompression-decompression formulas may be used. After decompression iscomplete, a volume of data 13 is then formed at step 60 of FIG. 1 bystacking the series of CT images 12 in computer memory. The formation ofthe volume of data 13 can be performed by using existing volumeformation programs, such as VoxelView™ (VitalImages, Fairfield, Iowa),or by using a customized program. Since each CT image 12 isapproximately 0.5 megabytes in size, 500 images equates to about 250megabytes. Therefore, a graphics computer 26 with sufficient memory,such as 256 Mbyte RAM, and adequate disc storage, such as greater than 1gigabyte, is required.

Since rendering speed is inversely proportional to the size of thedataset, it is often necessary to reduce the size of the dataset of thethree-dimensional volume 13 in order to perform three-dimensionalrendering in real time. The process of dataset reduction is shown atstep 65 of FIG. 1 and is illustrated in greater detail in FIG. 4. In thepresent application, the dataset is reduced from its original size, suchas 250 megabytes, to a reduced size, such as 5-10 megabytes to expeditethree-dimensional rendering. The dataset reduction may be partiallyaccomplished by reducing the pixel resolution, for example, from 16bits/pixel to 8 bits/pixel, as shown at step 67. The reduction in pixelresolution reduces the contrast in the final three-dimensional image byreducing the number of shades of gray from 2¹⁶ or 65,536 shades to 2⁸ or256 shades. In the field of radiology, the gray-scale shading of CTimages correlates to x-ray attenuation which is measured in Hounsfieldunits (HU), and which ranges in value from −1024 to +3072. Water isassigned a value of 0 HU, soft tissue falls between 20 and 200 HU,contrast enhanced blood is >125 HU, bones are >250 HU, and air is lessthan −300 HU. In specific applications, the entire range of Hounsfieldunits may not be needed. Accordingly, a selected region of the scale maybe used. For example, the upper and lower values of the region may beclipped prior to scaling. Since the regions above 300 HU and below −700HU do not contain useful information for specific applications inrendering colons and airways, only the region values between 300 and−700 HU are scaled to the 256 shades of gray. The scaling is linear.However, non-linear scaling could be used to emphasize a particularregion.

Further reduction of the dataset size can be effected by selecting asubvolume of the dataset to be displayed, as represented at step 68 ofFIG. 4. Generally, the colon is subdivided into the rectum, the sigmoidcolon, the descending colon, the splenic flexure, the transverse colon,the hepatic flexure, the ascending colon, and the cecum. A separatesubvolume may be assigned to selected subdivisions or sections of thecolon. As shown in FIG. 7, a selected subvolume 114 containing theentire colon and portions of the small bowel is specified by determininga set of minimum and maximum coordinates (X_(min), X_(max), Y_(min),Y_(max), Z_(min), and Z_(max)) such that the selected subvolume 114 isdefined by the selected coordinates. For example, vertex 125 indicatesX_(min), Y_(min), and Z_(min) and vertex 126 indicates X_(max), Y_(max),and Z_(max). Using subvolumes reduces the size of the dataset andenables separate three-dimensional renderings of each selectedsubvolume.

If further dataset reduction is desired, the dataset size can bedecreased at step 69 of FIG. 4 by reducing the spatial resolution of thethree-dimensional volume, for example, in the case of 500 images, from512×512×500 voxels to 256×256×250 voxels. Since, reducing the spatialresolution can blur the final three-dimensional image, decreasing thespatial resolution is generally not a preferred process for datasetreduction.

In general, the data reduction process 65 converts the initial volume oforiginal CT images into a transformed volume made of reduced CT images.The transformed volume is thereafter used to permit isolation of thetargeted organ or other area of interest. The reduction of the datasetsize may not be necessary, however, if the graphics computer 26 hassufficient memory and processing capability relative to the size of theinitial volume of original CT images to effect efficientthree-dimensional renderings.

In order to produce a three-dimensional rendering of only the colon, thecolon must be isolated from the volume of data by image segmentation, asgenerally represented at step 70 in FIG. 1 and as represented in greaterdetail in FIG. 5. Image segmentation can be performed before or afterdataset reduction. A volume file pertaining to a patient is selected atstep 71 of FIG. 1 and read at step 72 into the active RAM memory of thecomputer 26. An optional Sample Crop procedure 73 for subcropping and/orsubsampling the volume of data can be invoked to further reduce thevolume of the dataset if desired. The Sample Crop procedure 73 enablesthe user to crop the volume of data along the X-axis, the Y-axis and theZ-axis and further permits the user to subsample data, if desired, alongeach axis in order to further reduce the volume dataset. After theSample Crop procedure 73, an Orthoslice procedure, as represented atstep 74 in FIG. 5, is utilized to select a slice through the volume.More specifically, an orthogonal slice through the volume is taken alongthe axial plane (a cross-sectional (X-Y) plane through the bodyperpendicular to the Z-axis and to the patient's spine), the coronalplane (a side-to-side (X-Z) plane through the body normal to theY-axis), or a sagittal plane (a front-to-back (Y-Z) plane through thebody normal to the X-axis).

The particular orthogonal plane and the specific location of the sliceplane may be selected by the user. Preferably, the orthoslice isselected to pass through a complex portion of the anatomy. The selectedorthoslice is displayed at step 76 on the display monitor 28 of thegraphics computer 26. After the orthoslice is selected and displayed, athresholded version of the same image is also displayed on the monitor28. A threshold range is used at step 75 to define a range of x-rayattenuation values that represent the organ of interest. The region ofinterest may be the air column which designates the air and soft tissueinterface of the colon wall.

The specific value of each pixel in the orthoslice corresponds to aphysical property such as the x-ray attenuation at the location of eachsuch pixel. To effect thresholding of the orthoslice, each individualpixel or grid position in the orthoslice is compared, at step 75, to theselected threshold range to determine if each such pixel value lies inthe designated threshold range. If an input pixel from the orthoslicefalls within the thresholded range, such input pixel is set to aselected maximum value, e.g. a value of 255 corresponding to a whitecolor, in the thresholded image. Otherwise, the input pixel is set to aselected minimum value, e.g. a value of 0 corresponding to a blackcolor, in the thresholded image to indicate that such input pixel fromthe orthoslice falls outside the designated threshold range.

The threshold range can be deduced by the physician based on his/herexperience as well as a visual comparison of the orthoslice with thethresholded image. The thresholded image is displayed simultaneouslywith the orthoslice on display monitor 28, as indicated at step 76 ofFIG. 5, to enable the threshold range to be manually adjusted in realtime to produce a good match between the thresholded image and theanatomical detail of the orthoslice. As shown in FIG. 8, the orthosliceis one of the reduced CT images 112 through the volume and it isdisplayed alongside a corresponding thresholded image 115. Thethresholded image 115 is a single reduced CT image that has beensubjected to the acceptance criterion of the thresholding process. Thethreshold range is varied until the thresholded image 115 closelymatches the organ of interest in the orthoslice 112.

The threshold range thus obtained by thresholding the orthoslice is thenapplied globally to the volume of data at step 79 of FIG. 5 to create athresholded volume. An exterior surface of equal voxel value, or anisosurface 15, may be defined on the thresholded volume.

The isosurface 15 of the thresholded volume is then used as the basisfor generating a wireframe model 16 of the colon as set forth in FIG. 5.As illustrated in FIG. 9, a wireframe model 116 of the rectum portion ofthe colon is depicted in which the spatial resolution has been reducedfrom 512³ to 256³. The vertices of the wireframe model 116 define aseries of polygonal surfaces 136 that approximate the surface of theorgan of interest. Associated with each of the polygonal surfaces 136 isa vector which is perpendicular or normal to the surface 136. Variousalgorithms can be used for creating the wireframe model 116 including,but not limited to, marching cubes, dividing cubes, and marchingtetrahedrons. According to the marching cubes algorithm, the wireframemodel 116 of the colon is assembled by fitting a polygon or polygonsthrough each voxel that is deemed to sit astride the isosurface 15(i.e., through each voxel on the isosurface 15). The way in which thepolygon or polygons are situated within each voxel depends upon thedistribution of the vertices of the voxel that lie inside and outsidethe surface. It is generally assumed that there are 15 possibledistributions. The final position and orientation of each polygon withineach voxel is determined by the strength of the measured property at thevertices. The polygon or polygons are therefore an accuraterepresentation of the surface as it passes through the voxel.

Since subjecting the entire data volume to simple thresholding maysometimes over/under estimate the diameter of the colon and accentuateimage noise, region growing and edge detection methods may be employedfor segmenting the colon. A generalized region growing procedure 81 isset forth in FIG. 6. Region growing 81 can be used instead of thethresholding procedure to effect segmentation of the organ or selectedregion of interest.

In region growing, a seed voxel is selected at step 82 which lies withinthe organ or region of interest. At step 83, the seed voxel is set to afirst value (i.e., 255 for white). The value of a neighboring voxel isthen read at step 84. The value of the neighboring voxel is compared toa threshold range to determine if such neighboring voxel falls withinthe acceptance threshold range at step 85. If the neighboring voxelfalls within the acceptance range, the neighboring voxel is set to thesame first value as the seed at step 86. The process then returns to theseed voxel (i.e., neighbor-1) at step 87. A check is made, at step 88,to determine if all the voxels neighboring the seed voxel have beentested. If all the neighboring voxels have not been tested, anothervoxel neighboring the seed voxel is read at step 84 and processingcontinues as described above. If all of the neighboring voxels have beentested at step 88, a new seed type voxel is picked at step 89. The newseed voxel is a voxel which has been determined to lie within the regionof interest but whose neighboring voxels have not yet been tested atstep 85. Processing then continues at step 84, as described above.

Returning to step 85, if the neighboring voxel does not fall within theacceptance range, then the neighboring voxel is set to a second value(i.e., 0 for black), at step 61, thereby indicating that the neighboringvoxel lies at the edge of the region of interest. The process thenreturns to the seed voxel (i.e., neighbor-1) at step 87′. A check isthen made, at step 88′, to determine if all the voxels neighboring theseed voxel have been tested. If all the neighboring voxels have not beentested, another voxel neighboring the seed voxel is read at step 84 andprocessing continues as described above. If all of the neighboringvoxels have been tested, a new seed type voxel is picked at step 89′. Acheck is then made, at step 88″, to determine if all the voxelsneighboring the new seed type voxel have been tested. If all theneighboring voxels have not been tested, a voxel neighboring the newseed type voxel is read at step 84 and processing continues as describedabove. If all the neighboring voxels have been tested, processing stopsat step 62, thereby indicating that the region of interest is completelybounded by edge voxels. In this manner, the organ of interest can bedetected without subjecting the entire volume of data to thresholding.This technique is therefore capable of reducing the occurrence ofartifacts (e.g., air filled organs other than the organ of interest) inthe final three-dimensional rendering.

The geometry of the wireframe model 16 is stored at step 78 of FIG. 5.The wireframe model is stored in the form of a set of vertices andinterconnecting line segments that define the wireframe model 16. Thewireframe model 16 is then rendered into a three-dimensional image 17,as represented at step 80 in FIGS. 1 and 5. As illustrated in FIG. 10, athree-dimensional image 117 of the rectum section of the colon has beenrendered with reduced spatial resolution from the wireframe model 116depicted in FIG. 9. Images of three-dimensional objects (wireframemodels) in world coordinates (X, Y, Z) are projected to two-dimensionalscreen coordinates using ray-tracing techniques. Imaginary rays sentfrom a user's viewpoint pass through a viewing plane (referenced to amonitor screen) and into the object (wireframe model). If a rayintersects an object, the corresponding viewing plane pixel is paintedwith a color; if no intersection is found, the pixel is painted asbackground. The criteria used to stop a ray determines what value isprojected onto the viewing plane pixel. Surface rendering as used torender the image of the rectum shown in FIG. 10 projects the firstvisible voxel. Volume rendering projects a weighted average of allvoxels along a ray.

Animation of three-dimensional objects is achieved by rapidly displayingmultiple three-dimensional views of the volume. Surface renderingcalculations are fast enough to allow interactive manipulation (i.e.,virtual reality) of the volume. Additional lighting techniques enhancean object's features by altering the shade of color of each viewingplane pixel. For example, “shaded surface” images add an illusion ofdepth by coloring pixels closer to the camera viewpoint with lightershades. Pixels in a shaded surface image reflect the distance betweenthe anatomical structure and the user's viewpoint (not the originalvoxel values). Information on the relative “position” of the user and avirtual light source, along with the information about the wireframemodel 16, are used to appropriately shade the wireframe model 16 toproduce a realistic impression of the anatomy. The rendering process 80can be accomplished using a general purpose volume visualizationprogram, such as IRIS Explorer™.

The rendering step 80 occurs rapidly and interactively, thus giving theuser the ability to “fly” through the volume of data. The direction of“flight” is controlled by the computer mouse 27 through directionalpointing of the cursor, and the speed (both backwards and forwards) iscontrolled by pressing buttons on the computer mouse 27. The speed ofinteractive three-dimensional rendering produces a “virtual reality”environment and allows the user to examine the image data in a mannerthat is analogous to real endoscopy.

The path (camera coordinates) of each simulated flight can be recordedand used in a “playback” mode to retrace the flight path. Individualthree-dimensional scenes (views, images) may also be recorded (stored)on the computer like photographs. The geometric representation of thewireframe model 16 of the colon and the volume dataset used to createthe wireframe model are stored, at step 78 of FIG. 5, on digital audiotape (DAT) or, preferably, on read/write optical discs. Each simulated“flight” through the colon can be recorded on VHS videotape on a videorecorder 30 for archival purposes at step 90 of FIG. 1. Each flight maybe recorded for later review by, for example, gastroenterologists andsurgeons.

To achieve an adequate speed of rendering (flying speed) but stillpreserve the anatomical detail of the original CT data volume 13, it ispossible to navigate or “fly” through reduced three-dimensional imagerybased on a wireframe model 16 built from the reduced dataset volume.When the motion stops, however, the three-dimensional scene may beredisplayed with the highest resolution possible using the original CTimage volume 13.

In addition to allowing the user to “fly” through the colon, the methodof the present invention can be used to take the user on a guided tourof the colon in an “auto-pilot” mode of operation. In the “auto-pilot”mode, the user is moved at a preselected speed through thethree-dimensional representation of the colon along the center line ofthe lumen of the colon. The center line of the colon can be determinedin one of several ways. One method of determining the central paththrough the lumen of the colon is illustrated in FIG. 23. A seed point91 is selected which lies within the lumen of the segmented colon 117′″.The plane 92, passing through point 91 that has the minimum area 93 ofcolon dissection is determined and the center 95 of such area 93 iscalculated. A new point is then selected which lies 1 cm away fromcenter point 95 in a perpendicular direction relative to the surfacearea 93 in plane 92 as shown by arrow 94. A new plane of minimum areathat dissects the colon and passes through the new point is determinedand the corresponding center of that new area is calculated. Thisiterative process is continued until a central path connecting thecenter points is determined.

Alternatively, repetitive morphological “erosions” can be performed onthe segmented colon through an iterative process analogous to removing“layers” of voxels. As each layer is eroded, the iterative level atwhich voxels disappear in each such layer is recorded. The erosionprocess is continued until all of the voxels disappear. The central paththrough the colon is then constructed by connecting the highest valuedvoxels (i.e., the last voxels to disappear by erosion).

By determining the central path through the colon, a single oblique(reformatted) plane (gray scale image) that is perpendicular to thecentral path at a certain point can be displayed thereby allowingsimultaneous viewing of the wall thickness of the colon and surroundinganatomy and/or pathology during the automatic flight. The oblique planeis perpendicular to the central path but is usually oblique relative tothe three orthogonal planes.

The method of the present invention can also be used to display thecolon in a “split” open view 118, as shown in FIG. 12. Athree-dimensional external rendering of the rectum 117′ of the colon, asdepicted in FIG. 11, can be split along the center line to expose theinterior surface 119 of a selected half of the colon. This split displaymode is particularly advantageous when simultaneous viewing of extendedsectional lengths of the colon is desired or when viewing of the anatomymust be done with a computer that is not capable of rendering withreal-time speed (i.e., not allowing the user to “fly” through the colonat a sufficiently fast speed).

The “split” view can be generated in one of several ways. The firstmethod is to define a cutting plane that is disposed parallel to theplane of the screen of the monitor 28 such that all portions of objectspassing through the imaginary plane are made invisible as the objectsare pulled closer to the viewer and intersect the cutting plane.Defining a cutting plane is a standard part of Explorer's renderer(viewer). The three-dimensional object (colon) is rotated in space andmoved closer to the user (viewer). As the colon passes through thecutting plane, the half or portion of the colon closer to the userbecomes invisible but the distant half or portion, with its surfacedetail, is available for inspection.

Another method for producing a split view is to predefine subvolumes inaccordance with step 68 of FIG. 4 so that a selected sectional length ofcolon is contained in two separate subvolumes corresponding to the twoseparate halves of the sectional length of the colon. That is, twoseparate subvolumes can be created so that one subvolume shows one openhalf of the colon and the second subvolume shows the other open half.Defining the bounding points to thereby define a bounding subvolume maybe done visually by looking at orthoslices for representing thesubvolume. It is faster to manipulate the two-dimensional orthosliceimages rather than the three-dimensional volume. Once the boundingcoordinates (X_(min), Y_(min), Z_(min), and X_(max), Y_(max), Z_(max))are defined, then the subvolume representing one-half of a colon sectionis processed (rendered).

Another desired method for splitting the colon is to use an imaginarycutting plane (line) that passes through the colon perpendicular to thecentral path through the lumen of the colon. The cutting line ismaintained at a constant level parallel to the X-Z plane. All voxels onone side of the cutting line are made invisible (i.e., set to zero valueor black, since voxels in the segmented colon have been previously setto value 255 or white) and the white half of the colon is rendered.Then, the process is reversed and the other half of the colon isrendered. Alternatively, the wireframe model 16 may be split in halfprior to rendering.

An additional feature of the present invention is that an external mapview 137 of the selected organ can be displayed, as shown in FIG. 13, toassist the user if the user loses orientation or location during aflight through the lumen of the organ. FIG. 14 shows a three-dimensionalrendering 117″ from inside the colon. If the user requests the displayof a map view while inside the rendering 117″ of the rectum shown inFIG. 14, the external map view 137 of FIG. 13 will be displayed with anindicator 138 for indicating the position of the three-dimensionalrendering 117″ of FIG. 14. The indicator 138 can be a “+”, as shown inFIG. 13, or any other selected symbol. Preferably, the indicator 138 maybe in the form of an arrow which also indicates the line of sight withinthe three-dimensional imagery.

The method of the present invention is particularly well suited foridentifying polyps and growths that extend into the lumen of the colon.However, some precancerous growths are manifested as a subtle thickeningof the colon wall. The present invention can also be used to identifythe areas of thickening as well. This can be accomplished by calculatingand displaying a two-dimensional oblique plane perpendicular to thecentral axis of the colon whose location relative to the central pathcorresponds to the three-dimensional view at hand. Alternatively, asegmentation process could be used to isolate the colon wall instead ofthe air column within the colon to determine the wall thickness.Pursuant to a texture mapping procedure, thickened areas or other areasof pathology can be visually distinguished from the normal wall of thecolon in the three-dimensional imagery by displaying the polygonalsurfaces on the wireframe model that represent the thickened orpathological areas with a distinguishable color.

The method of the present invention can also be used to display atracheobronchial airway in three dimensions, in accordance with thegeneral methods described in connection with FIGS. 1 and 2 and with thesystem described in connection with FIG. 3. More specifically, a patientis initially prepared at step 40 by administering a nonionic intravenousbolus of iodinated contrast agent with a power injector to aid indistinguishing the blood vessels surrounding the tracheobronchialairway. After an appropriate time delay (approximately 70 seconds), thepatient is scanned at step 45 from the thoracic inlet to the lung baseto produce a series of two-dimensional images 12. The images 12represent at least one physical property associated with thetracheobronchial airway. This property may be, for example, the x-rayattenuation value measured with helical CT scanning. Preferably, thespacing between successive images 12 is approximately 1 mm to produceisocubic voxels.

The scans may be performed with a GE HiSpeed Advantage Helical CTScanner 22 during a single breath-hold acquisition which is completed inabout 30 seconds. The scanning parameters may consist of a 0.12 inch (3mm) x-ray beam collimation, 0.24 inch/sec (6 mm/sec) table speed (2:1pitch), and a 0.04 inch (1 mm) image reconstruction interval. As aresult of the beam collimation and the reconstruction interval selected,there is considerable (2 mm) overlap between successive images 12.Typically, up to about 200 images are obtained. The images are stored ina compressed format on a GE computer console 24 associated with thescanner 22.

The series of CT scans is then extracted at step 50 from the imagedatabase on the computer console 24 in compressed format. Eachcompressed image represents a 512×512 image of picture elements, orpixels, and each pixel is comprised of 16 bits of data (16 bits/pixel).Once the data has been extracted, the data is transferred at step 52over a fiberoptic network 25 to the graphics computer work station 26,such as the Silicon Graphics Crimson VGXT computer work station (150 MHzprocessor, 256 Mbytes RAM). The image files 12 are preferablytransferred in the compressed format and then decompressed at step 55 onthe graphics computer 26. The extraction and transfer steps areperformed by three program modules. The first module residing on thecomputer console 24 extracts the images one at a time from the imagedatabase and places each extracted image in a subdirectory on thecomputer console 24. In addition to the image files, a text filecontaining information about the patient and the type of case (i.e.,lung) is created. The second module, which resides on the graphicscomputer 26, is initiated every 5 minutes and transfers the patient'stext file and image files from the computer console 24 to the graphicscomputer 26 and deletes such files from the computer console 24. Thethird module, which resides on the graphics computer, is also initiatedevery 5 minutes and is interleaved with the second module. The thirdmodule determines if all of the files associated with a patient havebeen transferred. If all of the files have been transferred, the thirdmodule organizes the transferred files in a patient subdirectoryaccording to the case type. The entire process generally takes about 1hour and is a rate limiting step. The image transfer time can be reducedby utilizing the DICOM 3 image standard. Once the data transfer iscomplete, the remaining steps are performed on the graphics computer 26.

A volume of data 13 is then formed at step 60 by stacking the series ofCT images 12 in the computer memory. Since each CT image 12 isapproximately 0.5 megabytes in size, 200 images equates to about 100megabytes. Therefore, a machine with sufficient memory and adequatestorage is needed.

Since rendering speed is inversely proportional to the size of thevolume of data to be rendered, it is often necessary to reduce the sizeof the dataset in order to effectively perform three-dimensionalrendering in real time. Dataset reduction is generally shown at step 65of FIG. 1 and, more specifically, in FIG. 4. In application, the datasetis reduced from 100 megabytes to about 5-10 megabytes. Dataset reductionis partially accomplished by reducing the pixel resolution from 16 to 8bits/pixel, as represented at step 67. The reduction in pixel resolutionreduces the contrast in the final displayed images by reducing thenumber of shades of gray to 256. In the field of radiology, the grayscale shading of CT images corresponds to x-ray attenuation valuesmeasured in Hounsfield units (HU), which range in value from −1024 HU to+3072 HU. Water is assigned a value of 0 HU, soft tissue falls between20 and 200 HU, contrast enhanced blood is >125 HU, bones are >250 HU,and air is less than −300 HU. Since the regions above 500 HU and below−700 HU do not contain information useful for rendering the bronchialairways, or the surrounding blood vessels and lymph nodes, the regionbetween 500 and −700 HU is scaled to the 256 shades of gray. The scalingis linear. However, non-linear scaling could be used to emphasize aparticular region.

Further reduction of the dataset size can be accomplished by selecting asubvolume of the dataset to be displayed, as represented at step 68 ofFIG. 4. As illustrated in FIG. 15, a selected subvolume 214 is depictedcontaining the entire tracheobronchial airways. The bronchial airwaysare isolated from the entire thorax. The subvolume 214 is defined, asshown in FIG. 15, by determining a set of minimum and maximumcoordinates (X_(min), X_(max), Y_(min), Y_(max), Z_(min), Z_(max)) sothat the airways are contained within the subvolume 214 defined by theselected coordinates. For example, vertex 225 indicates coordinateX_(min), Y_(min), and Z_(min) and vertex 226 indicates coordinateX_(max), Y_(max), and Z_(max).

The dataset size can be reduced further at step 69 of FIG. 4 bydecreasing the spatial resolution, for example in the case of 200images, from 512×512×200 voxels to 256×256×100 voxels. Since reducingthe spatial resolution can blur the final displayed images, decreasingthe spatial resolution is generally not preferred.

The data reduction step 65 functions to convert the original volume ofCT images into a volume of reduced CT images. The volume of reduced CTimages is then used to construct the three-dimensional imagery.

As represented at step 70 of FIG. 1, the bronchial airways are isolatedwithin the volume of data by image segmentation. Image segmentation canbe performed before or after dataset reduction. To effect imagesegmentation, the volume file pertaining to the patient and case ofinterest are selected at step 71 of FIG. 5 and read at step 72 into theactive RAM memory of the computer 26. An optional subcropping and/orsubsampling step 73 can be performed, if desired, to further reduce thesize of the dataset. A threshold range is used to define a selectedrange of x-ray attenuation values contained within the organ ofinterest. Typically, the region of interest is the air column whichdesignates the air and soft tissue interface of the bronchial wall.

At step 74, an orthoslice is selected through a complex portion of theanatomy. The orthoslice is a cross-sectional two-dimensional slice(image) through the dataset parallel to one of the primary planes (X-Y,X-Z, or Y-Z). The orthoslice is displayed on the monitor at step 76 asone of the reduced CT images through the volume. Alongside the reducedCT image, a thresholded image is also displayed at step 76. Thedisplayed thresholded image represents the orthoslice of the reduced CTimage that has been subjected to an acceptance criterion of thethresholding process as previously described. As shown in FIG. 16, anorthoslice image 212 of the airways is simultaneously displayed with acorresponding thresholded image 215. The thresholded image 215 in FIG.16 is shown with the region of interest, the air column within theairways, in black. The threshold range is varied until the displayedthresholded image 215 best matches the anatomical detail of the organ ofinterest in the displayed gray scale orthoslice image 212. The thresholdrange thus obtained is applied throughout the selected volume of data tocreate a thresholded volume representing the airways. As an alternativeto thresholding, a region growing procedure, as previously described andas represented in FIG. 6, may be employed to isolate the organ or regionof interest. The region growing procedure is able to reduce theoccurrence of artifacts (i.e., air filled structures other than theorgan of interest) in the three-dimensional imagery.

An isosurface 15 of the thresholded volume is then used as the basis forforming a wireframe model 16 of the bronchial airways. As shown in FIG.17, a wireframe model 216 of the tracheobronchial airways is depicted.The vertices of the wireframe model 216 define a series of polygonalsurfaces 236 that approximates the surface of the organ of interest.Various algorithms can be used for creating the wireframe model 216including, but not limited to, marching cubes, dividing cubes, andmarching tetrahedrons.

The wireframe model 216 is rendered at step 80 of FIG. 1 into aninteractive, three-dimensional display 217, as illustrated in FIG. 18.The rendering procedure can be accomplished using a general purposevolume visualization program, such as IRIS Explorer™. The renderingprocedure gives the user the ability to “fly” through the volume ofdata. The direction of “flight” is controlled by the orientation of thecomputer mouse. The speed of the “flight” (both backwards and forwards)is controlled by pressing computer mouse buttons. Information on therelative “position” of the user and a virtual light source, along withthe information from the wireframe model 216, are used to appropriatelyshade the wireframe model 216 to give a realistic impression of theanatomy. Interactive three-dimensional rendering creates a perception of“virtual reality”, and allows the user to examine the CT data in a waywhich is analogous to real bronchoscopy.

Each “flight” through the bronchial airways can be recorded on VHSvideotape at video recorder 30, for archival purposes, as represented atstep 90 of FIG. 1, and for later review by, for example, bronchoscopistsand surgeons. In addition, each flight through the airways can berecorded on the computer 26 by storing the path of movement as a set ofcoordinates reflecting the flight path through the three-dimensionalimagery. For example, the appropriate coordinates of the path ofmovement may be stored in computer memory as the path of movement isgenerated. The coordinates of the path of movement may be stored in asetting file on the computer 26 so that the path of movement may bereproduced (replayed) on the computer 26 at a later time. Individualthree-dimensional scenes (views, images) may also be recorded (stored)on the computer like photographs. The wireframe computer model of thebronchial airways can also be stored as represented at step 78 in FIG. 5on digital audio tape (DAT) or, preferably, on read/write optical discs.

As an additional feature, selected “go to” points (three-dimensionalviews) may be generated and stored during a flight through the lumen ofthe organ. A “go to” procedure, generally designated 130, is representedin the flow chart of FIG. 25. During a flight through the organ,selected points of interest, such as three-dimensional scenes or views,may be recorded and assigned, at step 131, to buttons appearing on ageneral diagrammatic map of the particular organ being examined. The “goto” points on the map are stored, at step 132, in an appropriate settingfile on the computer 26. In order to use the “go to” points, the userrequests the display of the map of the organ at step 133. The “go to”points (buttons) appear on the map of the organ so that the user canmove the mouse cursor to a selected button, at step 134, and then clicka mouse button so that the displayed three-dimensional view istransformed to the chosen view at step 135. In application with thecolon imagery, separate “go to” points may be assigned to respectivesections corresponding to the rectum, sigmoid colon, descending colon,splenic flexure, transverse colon, hepatic flexure, ascending colon andcecum. For the tracheobronchial airways, separate “go to” points may beassigned to respective sections corresponding to the trachea, carina,right upper lobe bronchus, right middle lobe bronchus, right lower lobebronchus, left upper lobe bronchus, and left lower lobe bronchus.

The system 20 also provides the user with the ability to display thethree orthogonal slices (along the axial, coronal, and sagittal axes)which pass through any selected “pick point” in the three-dimensionalimagery. This is accomplished by pointing the computer mouse cursor atthe desired location in the three-dimensional imagery and “clicking” onthat point with a computer mouse button. The point of intersection withthe first visible surface in the three-dimensional imagery is captured.The three-dimensional coordinate (X,Y,Z position) of the point ofintersection is used to calculate the three orthogonal planes (axial,sagittal, coronal) that pass through that point. The axial plane isparallel to the X-Y axes, the sagittal plane is parallel to the Y-Zaxes, and the coronal plane is parallel to the X-Z axes. As shown inFIG. 26, the three orthogonal planes passing through a selected pickpoint in the tracheobronchial airways are displayed in three separatewindows (140, 141, and 142) together with a main window 143 containingthe three-dimensional image 217′″ of the tracheobronchial airways.Cross-hairs are overlaid on each orthogonal plane image indicating theposition of the pick point. By moving the mouse cursor over one of theorthogonal images, the cross-hairs in that image follow the position ofthe cursor. As the position of the cross-hair changes, the other twoorthogonal images and their cross-hairs change to reflect the newposition of the point in three-dimensional space. The three-dimensionalimage in the main window remains frozen during this time and does notchange. The usefulness of this function is that it allows a physician toview and pan through the reduced volume CT image data that surrounds thepick point. In using this process, wall thickness at a selected locationmay be determined. In addition, surrounding anatomical structure orpathology may also be viewed. In addition to “clicking” on a single pickpoint in the three-dimensional imagery, the user may activate a selectedcombination of mouse buttons while moving the mouse through thethree-dimensional imagery. The points of intersection are immediatelycalculated and the corresponding orthogonal planes are displayed. Thismouse dragging procedure is analogous to “dragging” you finger acrossthe surface of the anatomy and displaying the orthoslices interactively.

The method of the present invention also provides the ability to renderportions of a region of interest transparent or semi-transparent.Typically, the tracheobronchial airway can be rendered semi-transparentor completely transparent to allow the user to “see” beyond the wall ofthe airway and locate various anatomical structures, such as lymph nodesand blood vessels. This aspect of the present invention is particularlyuseful as a guide for Transbronchial Needle Aspiration (TBNA). Forexample, needle placement can be guided to a selected location throughthe tracheobronchial wall. Making the tracheobronchial wall transparentto reveal surrounding blood vessels and lymph nodes may serve to preventunintentional puncture of blood vessels and lymph nodes during needleinsertion.

Referring to FIG. 24, a procedure 100 for making a selected organtransparent for the purpose of revealing surrounding organs is depicted.After a volume of reduced CT images is generated at the selected area ofinterest, selected organs, such as the tracheobronchial airway, thesurrounding blood vessels and the surrounding lymph nodes, are segmentedfrom the volume of reduced CT images. At step 70′, the air columnrepresenting the tracheobronchial airway is segmented. At step 70″, theblood vessels are segmented and at step 70′″ the lymph nodes aresegmented. The order in which selected organs are segmented may bevaried. In addition, each organ may be segmented using a thresholdingprocedure or the region growing technique.

If thresholding is employed for organ segmentation, the air columnrepresenting the tracheobronchial airway may be assigned a range lessthan −300 HU. The soft tissue representing the lymph nodes may bethresholded in the range of 30 to 80 HU. The blood vessels may bethresholded according to the contrast agent contained within the bloodso that the blood vessels are thresholded in the range greater than 125HU.

After the appropriate wireframe models 16 are generated for therespective organs, the airways, blood vessels and lymph nodes arerendered together in the same three-dimensional scene at step 80′ ofFIG. 24 for display on the screen of the monitor. To better distinguishbetween respective organs, each wireframe model may be produced in aseparate color.

At step 101 of FIG. 24, one of the organs is selected for transparency.Then, at step 102, the degree of transparency is selected. The degree oftransparency may be selected by a sliding scale from 0 to 100%. Inapplication, the walls of the tracheobronchial airway may be selected tobe displayed transparently, therefore the view from within the lumen ofthe tracheobronchial airway enables the location of the surroundingblood vessels and lymph nodes to be seen.

The method of the present invention can also be used to display thetracheobronchial tree in a “split” view 218, as shown in FIG. 20. Athree-dimensional rendering 217′ of a selected section of thetracheobronchial tree, as shown in FIG. 19, can be split along thecenter line to expose the interior surface 219 of the tracheobronchialtree.

An additional feature of the present invention is that an external mapview 237 can be displayed as shown in FIG. 21. FIG. 22 shows athree-dimensional rendering 217″ from inside the tracheobronchial treeentering the left mainstem bronchus. FIG. 21 shows an external map view237 with an indicator 238 for indicating the position within thetracheobronchial tree from which the rendering 217″ in FIG. 22 is taken.The indicator 238 can be a “+”, as shown in FIG. 21, or any othersymbol, such as an arrow which also indicates the line of sight withinthe three-dimensional imagery.

The present invention is not limited to the use of images obtained fromCT scanners. The present invention works equally well with tomographicx-ray data, ultrasound, positron emission tomography, emission computedtomography, and magnetic resonance images. In addition, although thepresent invention has been discussed in reference to its usefulness inexamining the colon and the tracheobronchial tree, the present inventionis also useful in rendering three-dimensional images of otherpassageways and anatomical structures, such as arteries, veins, solidviscera, stomachs, bladders, bone, and ventricles of the brain.

It will be recognized by those skilled in the art that changes ormodifications may be made to the above-described embodiments withoutdeparting from the broad inventive concepts of the invention. It shouldtherefore be understood that this invention is not limited to theparticular embodiments described herein, but is intended to include allchanges and modifications that are within the scope and spirit of theinvention as set forth in the claims.

1. A method for interactively displaying three dimensional structuresfrom a three-dimensional volume of data representing at least onephysical property associated with a three dimensional body, comprisingthe steps of: a) isolating a selected region of interest from the volumeof data based on selected values of the physical property representingthe isolated region of interest, wherein the step of isolating theselected region of interest comprises a thresholding step fordetermining threshold values corresponding to the selected the physicalproperty to isolate the selected region of interest; and b) renderingthe isolated region of interest in an interactive three-dimensionaldisplay, wherein the thresholding step comprises a threshold selectionstep for selecting a threshold range corresponding to the selectedvalues of the physical property representing the region of interest anda threshold adjustment step for adjusting threshold values to establishthe threshold range for isolating the region of interest.
 2. The methodas recited in claim 1, wherein the threshold adjustment step comprises:a) an orthoslice step of providing an orthoslice through the volume ofdata; and b) a display step for displaying the orthoslice and acorresponding thresholded image of the orthoslice, so that the thresholdvalues can be adjusted while comparing the thresholded image to thecorresponding orthoslice.
 3. The method as recited in claim 1, whereinthe region of interest comprises an air-tissue interface.
 4. The methodas recited in claim 3, wherein the air-tissue interface comprises an aircolumn within a body organ.
 5. The method as recited in claim 1, whereinthe region of interest comprises an air-tissue interface.
 6. The methodas recited in claim 5, wherein the air-tissue interface comprises an aircolumn within a body organ.
 7. The method as recited in claim 1,comprising simulating movement along a center line through the isolatedregion of interest.
 8. The method as recited in claim 7, wherein thestep of simulating movement along the center line through the isolatedregion of interest of the three-dimensional display comprises the stepsof: a) selecting a seed point which lies within the region of interest;b) determining a plane of minimum area through the region of interestthat passes through the seed point; c) determining the center point ofthe region of interest that is dissected by the plane of minimum area;and d) selecting a point which is spaced from the center point in aperpendicular direction relative to the plane of minimum area.